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Abstract 

Since the publication of Shannon's theory of one terminal source coding, a num- 
ber of interesting extensions have been derived by researchers such as Slepian-Wolf, 
Wyner, Ahlswede-Korner, Wyner-Ziv and Berger-Yeung. Specifically, the achievable 
rate or rate-distortion region has been described by a first order information-theoretic 
functional of the source statistics in each of the above cases. At the same time sev- 
eral problems have also remained unsolved. Notable two terminal examples include 
the joint distortion problem, where both sources are reconstructed under a combined 
distortion criterion, as well as the partial side information problem, where one source 
is reconstructed under a distortion criterion using information about the other (side 
information) available at a certain rate (partially). In this paper we solve both of these 
open problems. Specifically, we give an infinite order description of the achievable rate- 
distortion region in each case. In our analysis we set the above problems in a general 
framework and formulate a unified methodology that solves not only the problems at 
hand but any two terminal problem with noncooperative encoding. The key to such 
unification is held by a fundamental source coding principle which we derive by ex- 
tending the typicality arguments of Shannon and Wyner-Ziv. Finally, we demonstrate 
the expansive scope of our technique by re-deriving known coding theorems. We shall 
observe that our infinite order descriptions simplify to the expected first order in the 
known special cases. 



1 Introduction 



Research into source coding began with Shannon's solutions to two one-terminal problems: 
In the first the source is reconstructed in a lossless manner [1] and in the other under a fidelity 
criterion [2]. Since then considerable effort has been directed at extending Shannon's results. 
After a hiatus of more than a decade, Slepian and Wolf made the next major breakthrough 
by finding the achievable rate region for multiterminal lossless coding [3]. Soon after, two 
important discoveries took place in quick succession: 1) Wyner [4] as well as Ahlswede and 
Korner [5] derived the achievable rate region for lossless coding of one source with partial 
side information; 2) Wyner and Ziv gave the achievable rate-distortion region for encoding 
of one source with complete side information at the decoder [6]. Quite a few interesting 
results have since been reported: Berger et al. derived an inner bound on the achievable 
rate-distortion region for encoding of one source with partial side information [7]. Kaspi 
and Berger solved a source coding problem where individual encoders cooperate in a certain 
manner [8] . Berger and Yeung derived the achievable rate-distortion region for a two terminal 
problem where one source is perfectly reconstructed whereas the other is reconstructed under 
a fidelity criterion [9] . At this point source coding research was unmistakably geared towards 
solving the famous open problem where a joint distortion criterion applies to both sources 
[10]. Zamir and Berger took the first step in this direction by solving the joint distortion 
problem under a high resolution assumption [11]. 

In this paper, we solve the above joint distortion problem in a general setting. Specifically, 
we derive an information-theoretic expression for the corresponding rate-distortion region as 
a functional of source statistics. Further, we identify the partial side information problem as 
a special case. Consequently, our joint distortion result solves the partial side information 
problem upon appropriate specialization. However, we instead present an equivalent solution 
that provides specific insight into the partial side information problem. In particular, the 
inner bound derived by Berger et al. [7] will be seen as a straightforward consequence of such 
specific solution. Although we solve two hitherto open problems, the main contribution of 
our work lies not in providing solutions to individual problems, but in formulating a solution 
methodology, which (as we shall exhaustively enumerate in the second paper of this two part 
communication [12]) applies to any source coding problem with noncooperating encoders and 
one decoder. At the heart of our method lies a fundamental principle of source coding that 
generalizes the typicality arguments of Shannon [2] and Wyner-Ziv [6]. To lend credence 
to the versatility of our theory, we first specialize our partial side information result to the 
case where side information is completely available at the decoder and derive Wyner-Ziv 
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theorem as a corollary. We further demonstrate the expansive scope of our technique by 
outlining on its basis the proof of four known coding theorems given by Shannon's rate- 
distortion theory [2], side information theory [4, 5], Wyner-Ziv theory (from first principle, 
unhke as a specialization) [6] and Berger-Yeung theory [9] . In the second paper of this two 
part communication [12], we shall show that our methodology extends to problems involving 
any number of sources. 

We organize the present paper in the following manner. In Sec. 2, we pose the joint 
distortion and the partial side information problems and give the respective coding theorems. 
We also derive Wyner-Ziv theorem by specializing the solution to the partial side information 
problem. In Sec. 3, we state and prove our fundamental principle of source coding. We 
devote Sees. 4 and 5 to the derivation of our coding theorems stated in Sec. 2. Known 
coding theorems are derived based on our technique in Sec. 6. Finally, Sec. 7 concludes the 
paper with a summary and the future scope of our work. 

2 Source Coding under Joint Distortion 

We begin our analysis by posing in Sec. 2.2 the two terminal joint distortion problem where 
the two dependent sources arc separately encoded and jointly decoded under a combined 
distortion criterion. An information-theoretic expression of the achievable rate-distortion 
region is presented in Theorem 2.1. In Sec. 2.3, we pose the partial side information problem, 
whose solution is presented in Theorem 2.2. The inner bound of Berger et al. [7] is then 
identified as a straightforward corollary of Theorem 2.2. We also show that, in the special 
case where the side information is completely available, Wyner-Ziv theorem [6] follows from 
Theorem 2.2. First we need some notation and the notion of strong typicality. 

2.1 Notation 

Throughout this paper we denote random variables by uppercase letters such as X, Y, Z, 
and their alphabets by corresponding script letters X, y, Z. All alphabets are finite unless 
otherwise stated. By H{X) and I{X;Y), denote entropy of X and mutual information 
between X and Y, respectively. Further, denote the k-th element of a sequence by x{k), the 
corresponding sequence by {x{k)} and the collection of all elements indexed by ki through 
/c2 by x{ki,k2). Also write a;" = x(l;n), x^{k) — x{n{k — 1) -|- l;nk) and x^{k-i;k2) — 
x{n{ki — 1) + l;n/c2). In addition, denote the closure of set A by A. Finally, define the 
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e-strongly (e > 0) typical set of X ~ p{x) by [10] 



e A*" : 



-N(x\x'^) -p(x] 
n 



< -r— for all X e X 



(2.1) 



where N{x\x"') denotes the number of occurrences of x in the sequence x". In this paper, 
we consider only strong typicality which will henceforth be mentioned as typicality. Conse- 
quently, we have (for sufficiently large n) 

Pr{X" ^ T/")(X)} < e (2.2) 

due to strong law of large numbers [10], where {X{k)} are drawn i.i.d. ~ p{x). Also if 
x"' G Te"'\x), then we call a typical sequence. In an analogous manner, the jointly 
typical set of a collection of random variables X = {Xi,X2, ...,Xm) is defined by (2.1) with 
X, X and X replaced hy X, x and X — Xi x X2 x ... x Xm, respectively. 



2.2 Joint Distortion 

Problem Statement: Let {Xi, X2) ~ p{xi,X2) be two discrete random variables and draw 
i.i.d. copies {{Xi{k), X2{k))} ~p(xi,X2). Encode {Xi,X2) using two encoder mappings 

/i : X^ ^ Z, 

J2 ■ ^2 ^2 

for some alphabets Zi and Z2, and decoding using decoder mapping 

g:ZixZ2^X^x X^ 
under a bounded distortion criterion 

d:X^x Xi ^ [0, dmax] 

defined on Xi x X2. We call the mapping triplet (/i, f2,g) a joint distortion code of length 
n and 

{xlxl)^g{h{X-,lh{X^)) (2.3) 

the estimate or reconstruction of [X^^Xl^]. A rate-distortion triplet (i?i,i?2,-D) is said to 
be achievable if for any e > there exists (for n sufficiently large) a joint distortion code 
ifi,f2,9) such that 

-log|Zi| < Ri + e (2.4) 
n 

-log 1 < i?2 + e (2.5) 

n 

-Edn{{X^,XS),{X;,X;)) < D + e (2.6) 
n 



where 

n 

- J]ci((xi(^),X2(^)),(xi(A;),X2(fc))). (2.7) 
fe=i 

Denote the set of achievable triplets ii!2, D) by (subscript 'D' indicating a distortion 
criterion involved). Clearly, Ao is closed. Our task is to express Ao as an information- 
theoretic functional of the source distribution p. 

Results: Let A^t^ be the set of (i?i, i?2, -D) triplets such that there exist alphabets Zi and 
Z2, conditional distributions qi{zi\xi) and q2{z2\x2), and mapping ip : Zi x Z2 ^ Xi x A"^, 
satisfying 

-I{X^-Z^\Z2) < Ri (2.8) 
n 

-I{X^;Z2\Z,) < R2 (2.9) 

n 

-I{X^,X^;Zi,Z2) < R1 + R2 (2.10) 
n 

-Ed^{{X^,X^),ij{Z,,Z2)) < D (2.11) 
n 

where 

(X-,X-,Zi,Z2) ~p„(x^x^)gi(^i|x^)g2(^2k^) (2.12) 

and Pn{x'^,X2) — 11^=1 P(^i(^)> ^2(^))- Note that (2.12) is equivalent to stating that Zi — > 
— > X2 — > ^2 forms a Markov chain. Further, let 

00 

= U -^nu- (2.13) 

n=l 

Note that each A^j^ is closed; however, A^ not necessarily is. 

Theorem 2.1 = 

Next we pose the partial side information problem considered by Berger et al. [7] . 

2.3 Partial Side Information Problem 

Problem Statement: Consider the problem where source Xi is encoded with side infor- 
mation X2 partially available at the decoder. The problem setting remains the same as that 
in Sec. 2.2 except that the distortion criterion d : — > [0, c?max] is now defined over Xi 
alone. Accordingly, a partial side information code consists of a mapping triplet 

(/i : X^ ^ Zi,/2 : X^ ^Z2,g:Z^xZ2^ X^). 
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Denote by the set of rate-distortion triplets {Ri,R2,D) achievable by such codes (ad- 
ditional subscript 'P' indicating partial side information). Any R2, D) e if for any 
e > there exists (for n sufficiently large) partial side information code (/i, /2, g) such that 
(2.4) and (2.5) hold alongside the distortion condition 

-Edn{X^,x'l)<D + € (2.14) 
n 

(in place of (2.6) seen in case of ^d)- Here X-^ = (yf(/i(X"), /i(X")) now. As earlier, note 
that is closed. We give an information-theoretic description of Aup in the following. 

Results: We can specialize Theorem 2.1 in an intuitive way to the partial side informa- 
tion problem where we have {d : Xl — > [0, dmax], 9 '■ Zi x Z2 ^ Xi). Specifically, define 
A!*^p in the same manner as ^* ^ except that the mapping is now of the form Z1XZ2 — > A"" 
and condition (2.11) is now replaced by 

-E4(Xr,V'(Zi,Z2)) (2.15) 
n 

Further, denoting = U^i ■^nDvi '^^ can show = vA'i^p (mimicking the proof of 
Theorem 2.1). According to this description, any (i?i, i?2, D) e Adp satisfies four conditions 
(2.8)-(2.10) and (2.15) for some n. 

We can also describe in an equivalent but more insightful manner using only three 
conditions. Let A*^^p be the set of {Ri, R2, D) triplets such that there exist alphabets Zi 
and Z2, conditional distributions qi{zi\x^) and ^2(^212^2)) mapping ip : Zi x Z2 —>■ A"", 
satisfying 

-I{X^;Z,\Z2) < Ri (2.16) 

-I(X^;Z2) < R2 (2.17) 
n 

-Edn{X^,iP{Z,,Z2)) < D (2.18) 
n 

where (X^", ^3", Zi, Z2) ~ p„(a:^ a;^)gi(zi|a;?)g2(^2k2)- Further, let 

00 

-^^P = U ^noP (2.19) 
n=l 

as earher. Note that each ^*dp is closed; however, not necessarily is. 



Theorem 2.2 A^p = A^p. 
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In view of definition (2.19), Theorem 2.2 impfies 2 A^j^p. This inner bound was 
derived by Berger et al. [7]. Moreover, if the side information X2 is completely available at 
the decoder, i.e., an additional constraint R2 — H{X2) is imposed, then it can be shown from 
Theorem 2.2 that the set ('C' in the subscript indicating complete side information) of 
achievable pairs D) is given by Wyner-Ziv theorem [6]. An outline of the proof is given 
below. Additional steps necessary to make the proof rigorous appear in Appendix A. 

Wyner-Ziv Theorem as Corollary: Clearly, the achievable set is given by 

A^c = D) : i?2, D) e ^„p, R2 = H{X2)}. (2.20) 

Also, define A*^^^^ by (2.20) with in place of ^*Dp. Further, defining Al,^ = U^i -4*^0, 
we have = ^dc ^o Theorem 2.2. 

Now let us inspect A^^^ closely. Noting ^7(X^; Z2) < \H{X^) = //(X2), condition 
(2.17) automatically holds for R2 — H{X2) and, as we shall see in Appendix A.l, one may 
assume Z2 — X2 with probability one without loss of generahty. Hence referring to (2.16) 
and (2.18), ^*dc is now described by the set of (i?i, D) pairs such that there exist alphabet 
Zi, conditional distribution qi{z]\x^) and mapping g : ZiX X2 — > A"", satisfying 

-I{X'l]Z-i^\X^) < Ri (2.21) 

-Edr,{X^,g{Z,,X^)) < D (2.22) 
n 

where (Xf,X^,Zi) Pn{x'l,x^)qi{zi\x'l). Further, referring to (2.21) and (2.22), recall that 
Wyner-Ziv theorem states = ^idc [6]- Consequently, since = ^dc by Theorem 2.2 
(as noted earlier), showing A^^. — Al^^^ amounts to proving Wyner-Ziv theorem. Moreover, 
noting ^ «4iDc froni definition, we only require Al,^ Q A^j^^- This indeed holds because 
AIj^c is closed and ^*dc ^ ^idc ^'^^ The last fact is shown in Appendix A. 2. 

Now we turn our attention to proving Theorems 2.1 and 2.2. Towards this goal we shall 
first derive in Sec. 3, a fundamental principle based on typicality that (upon appropriate 
generalization, as we shall see in [12]) governs the inner bound of achievable region arising 
in any source coding problem with noncooperating encoders and one decoder. Based on 
this principle, the proofs of Theorems 2.1 and 2.2 will then be presented in Sees. 4 and 5, 
respectively. 
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3 Fundamental Principle 



3.1 Statement 

Theorem 3.1 Let {Yi, Y2, Zi, Z2) be four random variables following joint probability distri- 
bution p'{yi,y2)q'i{zi\yi)q2{z2\y2) and draw {{Yi{k),Y2{k))} i.i.d. ^p'{yi,y2)- Then for any 
rate pair {R'l, R2) such that 

/(yi;Zi|Z2) < R[ (3.1) 

I(Y2;Z2\Z,) < R'2 (3.2) 

I{Y,,Y2;Z,,Z2) < R[ + R'2 (3.3) 

and any e' — > 0, there exists a sequence of mapping triplets 

(/i : 3^r' ^ Wi,/2 : y^' ^U2,g:UixU2^ Z^' x Z^') 

for some sequence of alphabet pairs (^1,^2) (and some n' — > ooj such that 

^log|Wi| < R'l + e" (3.4) 
n' 

^ logical < i?2 + e" (3.5) 
n' 

Pt{S} < e" (3.6) 



where 



^1 ^n' 



s = {(yf , Y2-' , z\ ,zl)i Ty\Y,, Y2, Z,, Z2)}, 



{Z, , Z2 ) = gUi{yr'), f2{Yi^')) and e" ^ 0. 

Note that Zi ^ Yi ^ Y2 ^ Z2 is required to form Markov chain for Theorem 3.1 
to apply. Further, due to strong law of large numbers, the sequence 

{(^".^2")} in the 

above can be replaced without loss of generality by any sequence {{Yl'\Yf)} such that 
Pr{(yi"',y2"') i T^'''\Yi,Y2)} < e[ where e[ as e' ^ 0. Such substitutions are standard 
and will sometimes be carried out without explicit mention. Also note that, as e' — >■ 0, 
n' ^ 00 through values n' > nQ(e') for some appropriate nQ(-). 
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Theorem 3.1 roughly states the following. Using a sequence of codes (/i,/2,5') (of suffi- 
ciently large length n'), one can achieve any rate pair {R[,R2) satisfying conditions (3.1)- 
(3.3) such that the estimate (ii',ij') = ^(/i (>?'), /2(1T')) of {Z^',Z^') based on the en- 
coding (/i(yi"'), 72(^2"' )) is jointly typical with {Y^' ,¥2') with high probability. Further, 
Theorem 3.1 includes the direct statement of Slepian-Wolf theorem as a special case. To see 
this, set {Zi, Z2) — (Yi, Y2) and note that the left hand sides of (3.1)-(3.3) reduce to H{Yi\Y2), 
H(Y2\Yi) and if (1^1,1^2), respectively. Now, for any rate pair [R'-^^R'^) satisfying (3.1)-(3.3), 
there exists a code {fij2,g) such that (3.4) and (3.5) hold and Pr{(yf ,^2 ) 7^ 
is small due to (3.6). This is the desired direct statement. 

Notice that Theorem 3.1 makes no reference to any distortion criterion. Yet we shall see 
that Theorem 3.1 plays a central role in proving the inner bounds in Theorems 2.1 and 2.2, 
each of which solves a problem where reconstruction is based on a distortion criterion. To 
give an insight into how Theorem 3.1 can be applied to a problem with a distortion criterion, 
let us sketch the proof of the inner bound arising in Theorem 2.1 (which we subsequently 
formalize in Sec. 4.1). To avoid technicality, let us show only Ao ^ A'^ for the time being. 
First of all, for any triplet {Ri, R2, D) e A*^ = Ur=i ^Id, (2-8)-(2.11) hold for some n. Now 
recaUing that Zi Z2 is Markov chain and identifying (^1,^2) = (Xf,X^'), 

note that (3.1)-(3.3) hold for = nRi and R2 = ni?2- Therefore, by Theorem 3.1, for any 
e ^ 0, (3.4)-(3.6) hold for some sequence of mapping triplets (/i,/2,5')- Note that (3.4) 
are (3.5) are identical to conditions (2.4) and (2.5) with {Ui,U2,nn') now playing the role of 
{Zi, Z2,n). We are only left to sec that the distortion given by the left hand side of (2.6), 
with nn' now in place of n, is close to D. For this, denote 

(XT', xT) = nzl, Zt) = (V'(ii(l), ^2(1)), V'(ii(2), ^2(2)), ^l^{Z,{n'l Z2{n'))) 

and observe that (Xf"', X^**') is jointly typical with il^'{Z-^ , Z2 ) with high probability due 
to (3.6). Consequently, in view of (2.11), the distortion requirement is met as desired. 

Next we prove Theorem 3.1 in three steps: The first two steps, given in Sees. 3.2 and 3.3, 
involve the derivation of two specializations. Lemmas 3.2 and 3.5, respectively. The third 
step, given in Sec. 3.4, completes the proof by combining the above lemmas. We shall see 
that Lemmas 3.2 and 3.5 capture the essence of the typicality arguments of Shannon [2] and 
Wyner-Ziv [6], respectively. 
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3.2 Derivation of Lemma 3.2 



Lemma 3.2 Let {Y, Z) ~ p'{y)q'{z\y) and draw {Y{k)} i.i.d. ~ p'{y) 
R' such that 

I{Y; Z) < R' 

and any e' — > 0, there exists a sequence of mapping pairs 

{f:y-'^U,g:U^Z^') 
for some sequence of alphabets U (and some n' —>■ oo) such that 

^log|W| < R' + e" (3.8) 

n' 

Pr{(y"',z"') ^rj"'^(r,z)} < e" (3.9) 

where Z = )) and e" 0. 

The direct statement of Shannon's rate-distortion theorem follows from Lemma 3.2 in 
a straightforward manner. Following gives an outline. Consider encoding of F ~ p'{y) 
under bounded distortion criterion d. Further, consider all conditional distributions q'{z\y) 
such that {Y, Z) ~ p'{x)q'{z\x) satisfies Ed{Y, Z) < D. By Lemma 3.2, one can construct 
block codes (/, g) of F"' that achieve rates close to I{Y; Z), while the decoded sequence Z 
remains jointly typical with with high probability. Hence, reconstructing Y = Z , the 

„ n' 

distortion AErf„/(F" ,r ) remains close to D. This is Shannon's direct theorem [2]. The 
above argument will be extended in Sec. 6.2. 

The above analysis demonstrates that the essence of Shannon's direct theorem is ab- 
stracted in Lemma 3.2 which makes no reference to the distortion criterion d. In fact, 
Lemma 3.2 will play a significant role in the development of a unified source coding theory. 
Proof of Lemma 3.2 is based on a random coding argument which borrows from the classical 
derivation of Shannon's direct theorem [10]. To proceed, we need intermediate results given 
in Lemmas 3.3 and 3.4. 

Lemma 3.3 1 — x < e'^ for x > 0. 

Proof: Denote f{x) — e~^ -\-x — 1. Differentiating, we obtain f'{x) — —e~^ + 1 > for 
X >0. Hence f{x) > /(O) = for x > and the result follows. □ 



Then for any rate 



(3.7) 
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Lemma 3.4 [10, Lemma 13.6.1] Let {Y, Z) ~ p'{y)q'{z\y) ^'^^ draw {Z{k)} i.i.d. ~ p'zi^) — 
T.y&yP'{y)(l'{Ay)- Then 

2-n'(7(y;Z)+el) < pj.|(^n'^ ^ ^K) (y^ z)} < 2-'(A^;^)-<) (3.IO) 

/or any y"'' e 7^1" '*(^); where e[ as e' ^ (and n' oo, appropriately). 

Proof of Lemma 3.2: Pick an integer K (a specific choice will be made later), draw 
each Z {l),Z {2),...,Z (K) i.i.d. ~ ULiP'zi^k) (where p'^(^) = E.ezP'ivWiAv)) in- 
dependent of F"', and define a random mapping F taking values in the set of mappings 
3^" ^ Z"- in the following manner: For any ?/" G 3^*^ , if there exists Z (i) such that 
(y" , Z (i)) e 7^; (l', Z), then set F{y^ ) = Z (i) (if there are more than one such Z (i), 
pick any one), otherwise assign F(f/^ ) to arbitrary Z (i). Of course, each mapping /, such 
that Pr{F = /} > 0, takes at most K values, i.e., for any such /, we have 

\U\<K (3.11) 

where U denotes the alphabet of f {¥"''). Also, for each y"' e T^^\y), we have 

Pr{(y"', F{y-')) i T^r'\Y, Z)} - f[{l - Pr{(y"', i"'(i)) i Tj'^^Y, Z)}) (3.12) 

i=l 

< (i_2--Viy;Z)+e',}Y (313) 

< exp (-2-"'(^(^'^)+<)x) , (3.14) 

where (3.13) follows by Lemma 3.4 (note — > as e' — > 0), and (3.14) follows by using 
Lemma 3.3 and setting x — 2~"''^^^^'^'>~^^'i\ Further, since 

Pr{(r"',F(r"')) e Ty\Y,Z)\Y"' ^ Tf\Y)} = 0, 

we have 

Pr{(F"',F(F"')) ^ Ty\Y,Z)} (3.15) 

= 1 - Pr{r'^' e Ty\Y)} Pr{(r"', F(r"')) e ry\Y, z)\y''' e ry\Y)} 

< 1 _ (1 _ e') [l - exp (-2-'(^(^;^)+^i)i^)] (3.16) 

< e' + exp (-2-^'inY;Z)+e',)j^^^ ^ ^/ ^ (3_17) 

say, where (3.16) follows due to strong law of large numbers and by (3.14). Now substitute 
the random mapping F in (3.15) by some mapping value / such that Pr{F = /} > and 
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upon substitution (3.17) still holds. Denoting the alphabet of by U as before, of 

course, (3.11) holds. Next choose K (while adjusting n', if necessary) such that 

I{Y; Z) + 2e[ < ^logK < I{Y; Z) + 3e[ (3.18) 

and, in (3.17), note that e'g — > as e' ^ by the lower bound in (3.18) and write 

e" = max{3e;,e2}. (3.19) 

Further, using (3.7), (3.11) and (3.19) in the upper bound in (3.18), we obtain (3.8). Finally, 
using (3.19) in (3.17) (with / now in place of F), (3.9) follows. □ 

3.3 Derivation of Lemma 3.5 

Lemma 3.5 Let {Yi,Y2, Zi) ^ p'{yi,y2)q'i{zi\yi) and draw {{Yi{k),Y2{k))} i.i.d. ^p'{yi,y2)- 
Then for any rate R[ such that 

I(Y,;Z,\Y2)<R[ (3.20) 

and any e' — >• 0, there exists a sequence of mapping pairs 

for some sequence of alphabets Ui ( and some n' — > oo j such that 

-\og\Ui\ < R[ + e" (3.21) 
n' 

Pr{(y-',y/,zf)^7;("')(yi,y2,^i)} < e" (3.22) 

where = g{fi{Yf), Y^') and e" 0. 

Wyner-Ziv's direct theorem follows from Lemma 3.5 in the similar manner as Shannon's 
direct theorem followed from Lemma 3.2. An outline is given in the following. Consider 
encoding of Yi under distortion criterion d assuming complete availability of side informa- 
tion Y2 at the decoder. Further, consider all conditional distributions q'i{zi\yi) such that 
(Yi,Y2,Zi) ~ p'{yi,y2)qi{zi\yi) satisfies E(i(Yi, ■?/'(Z2, 12)) < D for some mapping tp. By 
Lemma 3.5, one can construct complete side information codes {fi,g) that achieve rates R[ 
close to I{Yi; Zi\Y2), while the decoded sequence Z-^ is jointly typical with (Vj"', Y^') with 
high probability. Hence, Y{^' coupled with the reconstruction 

yf = (V'(ii(i),y2(i)),V'(^i(2),y2(2)),...,V'(^i(ri'),^2(n'))) 
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gives a jointly typical sequence of (li, ■0(Zi, 1^2)) with high probability, which ensures that 
the distortion ^Ed„/(y"'', ) is close to D. This is Wyner-Ziv's direct theorem [6]. 

The above analysis demonstrates that the essence of Wyner-Ziv's direct theorem is ab- 
stracted in Lemma 3.5 which makes no reference to the distortion criterion d. In fact, Lemma 
3.5 will play a significant role in the development of a unified source coding theory. Proof of 
Lemma 3.5 is based on a random coding argument which borrows from the classical deriva- 
tion of Wyner-Ziv's direct theorem [10]. To proceed, we need additional intermediate results 
given in Lemmas 3.6 and 3.7. 

Lemma 3.6 (1 — xY > I — kx for x e [0, 1] and k > Q. 

Proof: Denote f{x) = (1 — x)'^ — (1 — kx). Differentiating with respect to x, we obtain 
f'(x) = -k{l - xf'^ + > for a; e [0, 1]. Hence f{x) > /(O) = for a; e [0, 1], and the 
result follows. □ 

Lemma 3.7 [10, Lemma I4.8.I] Let {Yi,Y2,Zi) ~ p'{yi,y2)qi{zi\yi) and the sequence of 
triplets {{Yi{k),Y2{k), Zi{k))} be such that, for any e' — > (and an appropriate n' ^ 00), 

Y>,{{Y^\Y^')iTy\Y,,Y,)} < e\ 
Y>.{Y^\Z^')iry\Y,,Z,)} < e\ 

for some e'^ — > 0. Then 

Pr{(y,-',y/,^r') i ry\YuY,,z,)} < e', 

for some e'2 such that £3^0 as e' ^ 0. 

The above is a rewording of the so-called Markov lemma given in [10]. 

Proof of Lemma 3.5: Pick integers and K2 and draw each (1), (2), {Kx) 
i.i.d. ~ YYk=\Pz-i_{zi{k)) independent of (Yi ,Y2'), where 

pzi(^i)= ^ p'{yi,y2)q'i{zi\yi). 

{yi,y2)eyixy2 

Next randomize encoder mapping fi by defining a random mapping Fi over the set of 
mappings y^' — > in a two-step manner. First define random mapping Fi over the 
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set of mappings 3^"' — > Ik^ as follows: For any y"' e 3^"', if there exists z\ {i) such that 
(y"', (i)) e 7^1" ^(Fi, Zi), then set = i (if there are more than one such (i), pick 

any one and proceed); if there is no such (i), assign to arbitrary i e 7^^-^. Then 

defining a random variable (mapping) T such that T is independent of all preceding random 
variables and uniformly distributed over the set of mappings Ik^ — > /i^-j, set Fi(y"') = 
T{Fi{yl')). Clearly, for each /i such that Pr{Fi = /J > 0, the alphabet Ui of /i(yi"') is a 
subset of Ik2-i i-©-) 

\Ui\ < K2. (3.23) 

Correspondingly, randomize decoder mapping g by the random mapping G defined over 
^ 3^2*' ^1' following manner: For any j G Ji^j, if there exists unique (i) such 

that {y2', Zi (i)) G Tjl" ^(1^2, -Z^i) and T{i) = j, then set G{j, t/g') = -Z^i (Oi there is no such 
Z^ (i) or there arc more than one, assign G'(j, 1/2') ^'^ arbitrary Z-^ (i). 

Now note that the probability that {Y{^\ Y2') ^ot jointly typical is bounded by 

Pr{£o} = Pr{(Fi"', Yt) ^ T^'^iY^. Y^)] < e' (3.24) 

for sufficiently large n' . Next, the probabihty that Y2' is not jointly typical with Fi{Y{^') 
(i.e., any of the Z^^ (O's) is given by (recaUing the steps (3.12) through (3.17)) 

Pr{£i} = Vx{{Y^\ Fi{Yf)) i rj"'\Fi, Zi)} < exp (^-2-"'(^(^i=^i)+<)ii'i) + e' (3.25) 

where e'^ — > as e' — > 0. We shall make Pr{£^i} small by choosing K\ (while adjusting n', if 
necessary) such that 

/(Fi; Zi) + 2ei < ^ log < /(Fi; Zi) + 3ei. (3.26) 

n 

Using the lower bound in (3.25), we have 

Pr{£:i} < exp (^-2"'^'i) + e' (3.27) 

which approaches zero as e' — * (and n! 00, appropriately). Further, noting (3.24) 
and (3.27) and applying Lemma 3.7 (Markov lemma), the probability that {y^'^Y^') is not 
jointly typical with Fi(F"')) is bounded by 

Pr{£:2} = {(n"', IT', i ry\Y,, Y2, Z,)} < e'2 (3.28) 

where £2 ^ as e' ^ (and n' — > 00, appropriately). 

Next we bound the probability that there are more than one i's such that r(i) =Fi(y]") 
and {Y2' iZ^ (i))) are jointly typical. Define random variables Tj = T'^{j), j G Ik-2- Each 
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Tj takes values / C and follows identical distribution because T is uniformly distributed 
over all mappings Ik^ — Ik2- Now, for any 63 > 0, make n', Ki and K2 sufficiently large, 
keeping K1/K2 constant (e.g., by taking Ki and K2 in tandem through multiples), such that 

^ < e', (3.29) 



E 



which is possible due to strong law of large numbers. Without loss of generality, choose eg 
such that £3 — >• as e' — > 0. Further, by Lemma 3.4, we have, for any 1/2' G 7^/" \Y2), 



Pr{(yr,i"(i)) e Ty\Y2,Z^)} < 2--V(yr,z.)-e',) ^ 



(3.30) 



say, where €4 — > as e' — > (and n 00, appropriately). Moreover, for a given T'{j) = I, 
we have 

Pr {|{i e / : (y2"',^f (i)) e Ty\Y2,z,)}\ > 1} 

<l-(l-x)l^l-|7|x(l-x)l^l-^ 

<!-(!- \I\x) - \I\x{l - (|/| - l)x) (3.31) 
< |/|V, (3.32) 

where (3.31) follows from Lemma 3.6. Denoting 

£s = e T-i(Fi(F«')) : (F/,^f (z)) e Tj"')(F2, ^1)} | > l} , (3.33) 

replacing |7| by the corresponding random variable |T"(_7)| in (3.32)) and taking expectation, 
we obtain 

Primal < E[|T'|>2 < r^a-^-'^^^^^'^i)-^^) + j2-^n'(i(Yr,z,)-e',) ^3 34) 

which follows from (3.29) and by expanding x — 2~"'(^(^2;-^i)-el) shall make Prl^s} small 
by choosing (while adjusting n', if necessary) 



7(1^2; ^1) - 3^4 < ^ log K,-^ log K2 <I{Y2] Z,) - 2e'^. 



Specifically, we have 



PriSs} < exp (^2-2"'^4^ + e^2-2"' 



{I(Y2;Zr)-e'^) 



(3.35) 



(3.36) 



which approaches zero as e' ^ (and n' 00, appropriately). 

At this point, referring to (3.24), (3.25), (3.28) and (3.33), verify that the overall error 
event £ satisfies 



S = {{Yr',Y2^',G{F^{Yr'),Y2^')) i T^r'\Y^, Y2, Z^)} CSoUSiUS2U£s 



(3.37) 
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such that 

Pr{£:} < Pr{£:o} + Pr{£:i} + PriS^} + Pt{£s} < e'5 (3.38) 

where e'g — as e' ^ (and n' 00, appropriately) due to (3.24), (3.27), (3.28) and (3.36). 
Further, substitute the random mapping pair {Fi, G) in expression (3.37) of £^ by a mapping 
pair value (/i, g) such that Pr{(Fi, G) = (/i, (?)} > and upon substitution (3.38) stiU holds. 
Denoting the alphabet of fiiY^') by Ui as before, of course, (3.23) holds. 

At this point, using the upper bound in (3.26) in the lower bound in (3.35) and noting 
I{Yi- Zi) - I{Y2; Zi) = I{Yi; Zi\Y2) (because Zi^Yi^ Y2 is Markov chain), we have 

^ log < /(Fi; Z1IF2) + 3e; + 3e^. (3.39) 
n' 

Now set 

e" = max {3e[ + 3e^, e'J (3.40) 

(of course, e" ^ as e' ^ 0). Using (3.20), (3.23) and (3.40) in (3.39), we obtain (3.21). 
Finally, using (3.40) in (3.38) (with {f,g) now in place of {F,G)), (3.22) follows. □ 



3.4 Completion of Proof of Theorem 3.1 

First, note 

I{Y,;Z,\Z2) = H{Zi\Z2)-H{Zi\Y,,Z2) 

= H{Z,\Z2)-H{Z,\Y,) (3.41) 

< H{Z,)-H{Z,\Y,) (3.42) 

= /(Yi;Zi) (3.43) 

where (3.41) follows because Zi ^ Yi ^ Z2 is Markov chain and (3.42) follows because 
conditioning reduces entropy. Further, apply the chain rule and note 

I(Y,,Y2;Z,,Z2) = I(Y,,Y2;Z,)+I{Y,,Y2;Z2\Z1) 

= I{Y,-Z,)+I{Y2;Z,\Y,) + I{Y2-Z2\Z,) + I{Y,-Z2\Z,,Y2). (3.44) 

Since Zi ^ Yi ^ Y2 ^ Z2 is Markov chain, we have 

I{Y2:Zi\Yi) = (3.45) 
I(Y,;Z2\Z,,Y2) = H{Z2\Z,,Y2)-H{Z2\Z,,Y2,Y,) 

= H{Z2\Y2)-H{Z2\Y2)^0. (3.46) 
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Consequently, we conclude 

/(Fi, Y^- Zi, Z2) = I{Y,- Z,) + !{¥,■ Z,\Z,) (3.47) 

using (3.45) and (3.46) in (3.44). 

At this point, denote by B* the set of {R[, E!^ satisfying (3.1)-(3.3). Clearly, B* is convex. 
Further, we conclude the following from (3.43) and (3.47): 1) Any (i?'/°\ i?^^°^), such that 

> I{Y,-Z,) (3.48) 

> /(F2;^2|^i), (3.49) 
belongs to B*; 2) Any {R[^^\ R'^^^^) , such that 

i?'/'^ > /(Fi;Zi|Z2) (3.50) 

> 1(^2; ^2), (3.51) 

also belongs to B* , by symmetry; 3) Any {R[,R'2) G B* can be expressed as a convex 
combination of (i?'/°\ 7?^^°^) and {R[^^\ R'^^^^) such that (3.48)-(3.51) hold. 

Further, denote by B the set of rate pairs {R[,R'2) such that for any sequence e' — > 
there exists mapping triplet (/i,/2,5') satisfying (3.4)-(3.6). We claim that B is convex. 
To see this, note that if each of two rate pairs {R[^^\ R'2^^^) and {R[^^\ Ri^^^^) belongs to 
B, by a time sharing argument, so does any convex combination of such pairs. Clearly, 
to complete the proof of Theorem 3.1, we need to show B* C B. Towards this goal, we 
claim that any {R'i^^\ -^2^°^), satisfying (3.48) and (3.49), belongs to B. To see this, note the 
following. Firstly, by Lemma 3.2, for any e' — > 0, there exists a sequence of mapping pairs 
(/i : y?' Wi, gi-.Ui^ Zf) such that 

lloglWil < Rf^ + e[ (3.52) 

n 

Pr{{Y,^',z''^)^ry\Y^,Z^)} < e[ (3.53) 

where Z^ = gi{fi{Y{'')) and e[ ^ as e' ^ 0. Also, in view of (3.53), there exists, by 
Lemma 3.5, a sequence of mapping pairs (/2 : 3^2 ' ~^ ^2, 92 -^^2 ^ Z'f Z^) such that 

1 log 1^2 1 < i?2^°^ + 6'2 (3.54) 

n 

VT{{Y^\ztzt)ir^r'\Y^,Z^,Z^)} < e'2 (3.55) 

where Z^ = 92{f2{Y^''), ^f) = 92{f2{Yf), gi{h{Y{''))) and e'^ ^ as e' ^ 0. Noting (3.53), 
(3.55) and the fact that Zi ^ Yi ^ Y2 ^ Z2 is Markov chain, apply Lemma 3.7 (Markov 
lemma) repeatedly to obtain 

Pr{(y,"', y/, , Z2) i Ty\Y,, ^2, ^1, Z2)} < (3.56) 
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where 



(^1^2') =5(/l(n"'),/2(F/)) = (5l(/l(n"')),52(/2(F/),^l(A(n"')))) 

and €3 ^ as e' ^ 0. By (3.52), (3.54) and (3.56), respectively, and setting e" = 
max{e;,e'2,e;5}, (3.4)-(3.6) hold. Hence {R[^^\ R'^^^^) G B. By symmetry, any {R[^^\ R'^^^^), 
satisfying (3.50) and (3.51), also belongs to B. 

Now, recalling that any [R[, R'2) G B* can be expressed as a convex combination of some 
(i?'/°^ i?^^°^) and {R[^^\ R'^^^'^) such that (3.48)-(3.51) hold and noting that such pairs satisfy 
{R'i^\ R^^'^^) G B and {R!i^\ R!2^^^) G B (by the argument given in the last paragraph), we 
have (i?i, R'2) G B due to convexity of B. Hence B* C B and the proof is complete. □ 



4 Proof of Theorem 2.1 

Now we turn to the proof of Theorem 2.1 which consists of two parts: The inner bound 
A-D ^ A-^ is shown in Sec. 4.1 using the fundamental principle given in Theorem 3.1. The 
outer bound An Q is shown in Sec. 4.2 with the aid of Slepian-Wolf theorem [3]. 



4.1 Inner Bound A.^, ^ >A.^ 

For any triplet {Ri,R2,D) e A'^ — U^i^nD) (2-8)-(2.11) hold for some random variables 
(^1,^2) such that Zi — > X2 — > is a Markov chain and for some mapping ^0 : 

Zi X Z2 ^ Xi X X2- Referring to Theorem 3.1, note that conditions (2.8)-(2.10) are same 
as conditions (3.1)-(3.3) upon identifying {Yi, Y2, Zi, Z2) with {X^, X^, Zi, Z2) and (i?'i, i^'s) 
with (ni?i,ni?2)- Consequently, by Theorem 3.1, for any e' — > 0, there exists a sequence of 
mapping triplets 

(/i : ^ Uu /2 : Xf ^ U2, 9 : x U2 ^ Z^' x Z^') 

(for some n' 00) such that (3.4)-(3.6) hold. In other words, we respectively have (the first 
two conditions (3.4) and (3.5) are divided throughout by n) 

^log|Wi| < Ri + e"/n (4.1) 
nn' 

^log|W2| < R2 + e"/n (4.2) 
nn' 

Pr{£} < e" (4.3) 
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where 



£ = {{xr , xr ,z,,z,)^ vr\x^, x^, z,, z,)}, 

(zf , Zj') = c/(/i(Xr')> /2(^2""')) and e" 0. Further, denote 

{Xl{3),xl{j)) = i;{Z,{j),Z,{j)), l<j<n' 

{XT', XT') = ^l^\zt Zi) = g\h{Xf\h{XT')). 
Note that g' •.U1XU2 ^ A'f"' x X^'^'. Due to (4.3), we have (adjusting n' 00, if necessary) 

Pr{£i} < e'i (4.4) 

where 

= {{xr',xr',nz^,z;')) i rf'\xi,xi,^{z^,z2))) 

and — > as e' ^ 0. Also, due to (2.11), we have 

—d^^,{{xT\ xT'\ {xf: £T')) <D + e'dmax 
nn' 

for any {xY'',xY'', {£'"'', x^'')) G Tj"'^(Xf , ^(Zi, Z2)). Hence we obtain 

4n'((^r', ^r'), (^r', ^r')) < (1 - Pr{^i})(^ + e'c^max) + Pr{^i}o?^ax 

< D + {e' + e';)d^^ (4.5) 

using (4.4). Now, for a particular e > 0, choose e' > such that max{e", (e' + e'/)(imax} < 
Consequently, conditions (4.1), (4.2) and (4.5) give rise to conditions (2.4)-(2.6), respectively, 
with (V(i,U2, g', nn') now playing the role of (Zi, Z2, g, n). Hence {Ri, R2. D) G A^. In other 
words, Ay) ^ A^. Since Aj^ is closed, we have Ay^ ^ ^* (noting ^* is the smallest closed set 
with ^* as a subset). This completes the proof. □ 



4.2 Outer Bound A, C A* 

Recall that any triplet (i?i, R2,D) G if, for any e > 0, there exists joint distortion code 

(A : ^ Z^,f2 : ^ Z2, g : Z, x Z2 ^ x ^"2") 

of sufficiently large length n such that (2.4)-(2.6) hold. For easy reference, the above condi- 
tions are reproduced below: 

-log|Zi| < i?i + e (4.6) 
n 

-logical < i?2 + e (4.7) 

n 

-mn{{X^^,X^),g{h{y^l),j2{X^))) < D + e. (4.8) 
n 
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As seen in [6], we can further encode {Zi, Z2) — /2(X2 )) using an interposed 

Slepian-Wolf code 

Given {Ri, R2, D) G ^d, e and (/i,/2,fi'), a rate pair {R[, R2) is said to be achievable using 
interposed codes if for any e' > there exists mapping triplet {f[,f2,g') (of length n') such 
that 

^log|Wi| < R[ + e' (4.9) 

Th 

^ logical < R2 + e' (4.10) 

pr{(zrur)^^'(/;(^r'),/2«))} < e'. (4.ii) 

In view of (4.6) and (4.7), setting each of f[, f2 and g' to identity mapping (clearly, n' = 1, 
Ui = Zi,U2 = Z2), of course, (4.9)-(4.11) trivially hold for {R[, R'^) = (n(i?i + e), n(i?2 + e)) 
irrespective of e'. Therefore, we have 

^H{MXW2iX^)) < Ri + e (4.12) 

-H{f2{X^)\MX^)) < R2 + e (4.13) 
n 

-H{h{X^)j2{X^)) < Ri + R2 + 2e (4.14) 

by Slepian-Wolf theorem [10] . 

At this point define for any e > and any integral n > 1 the set ^^d^ of rate-distortion 
triplets (i?i,i?2,-D) such that there exist alphabets Zi and Z2, conditional distributions 

qi{zi\x^) and ^2(^212^2)) mapping ip : Zi x Z2 ^ x X2-i satisfying 



-7(Xf;Zi|Z2) < i?i + e (4.15) 

n 

-I{X^;Z2\Zi) < R2 + e (4.16) 
n 

-I{Xl\X^';Zi,Z2) < i?i + i?2 + 2e (4.17) 

n 

-ci„((Xr,X-),V(^i,^2)) < D + e (4.18) 



where {X^,Xl\ Z^, Z2) ~ Pn{x^,x^)q-,{z-,\x^l)q2iz2\x^i) (i.e., ^ ^ ^2^^ ^ Z2 is a 
Markov chain) and p„(a;", Xg) = HILi P(^i(^'")! ^2(^))- Comparing (4.15)-(4.18) with (2.8)- 
(2.11), note that Ano^ = A*^^. In addition, Xd'^ C Xd'^ for ei < €2. Further, let Au^'^ = 
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U^i-^nD . Of course, Q An for ei < 62- Hence, noting ne>o'^D cfosed, we 

obtain 

n^ = ^ = :4^- (4.19) 

The second equality in (4.19) holds because = 0^=1 = \Jn=i = -^n- 

Finally, consider any {Ri,R2,D) G Recall that for any e > there exists joint 

distortion code {fi,f2,g) such that (4.12)-(4.14) and (4.8) hold. Choosing (^1,^2) = 
(/i(X"), /2(X^)) (of course, Zi X" Z2 is then a Markov chain) and ip = g, note 

that the above four conditions coincide with (4.15)-(4.18), respectively. Hence (i?i, R2,D) G 
A^n ■ Consequently, we have (i?i,i?2,-D) G ^d^'' C ^d''-' for each e > 0. Hence, by (4.19), 
we have (i?i, R2, D) G n€>o'^D''^ = ^d- This completes the proof. □ 

5 Proof of Theorem 2.2 

The proof is similar to the proof of Theorem 2.1 and consists of two parts again: The 
inner bound ^ is shown in Sec. 5.1 using Lemmas 3.2 and 3.5 (both specializing 
Theorem 3.1). The outer bound ^ -4.*p is shown in Sec. 5.2 using Shannon's lossless 
coding theorem [1] and Slepian-Wolf theorem [3] . 



5.1 Inner Bound «4dp ^ 

Since = Ur=i K^v, for any triplet (i?i, i?2, D) G ^^p, (2.16), (2.17) and (2.18) hold for 
some ip : Zi X Z2 ^ and random variables {Zi, Z2) such that Zi — > — > X2 — > Z2 
is a Markov chain. Referring to Lemma 3.2, note that (2.17) is same as (3.7) for the choice 
(Y, Z) — (X", Z2) and R' — nR. Consequently, by Lemma 3.2), for any e' — > 0, there exists 
a sequence of mapping pairs 

{f2:Xr' ^U2,g2-M2^Z^') 

(for some n' — > 00) such that (3.8) and (3.9) hold. In other words, we respectively have 
((3.8) is divided throughout by n) 

^log|W2| < R2 + e"/n (5.1) 
nn' 

Vi{{Xr\zt)iTf\x-,Z2)} < e" (5.2) 
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where tT^ = g2{f2{X^"'')) and e" ^ as e' ^ 0. Noting (5.2) and the fact that 
Z-i is a Markov chain, we have, by Lemma 3.7, 

pr{(xr', xr', K) i ^j"'Hxr, ^2 , z^)} < a (s-s) 

where — as e' ^ 0. Further, referring to Lemma 3.5, note that (2.16) is same as 
(3.20), where (X", Z2, Zi) (note Z\ X^ — » Z2 is a Markov chain) now plays the role 
of {Yi,Y2,Zi) and R'l — nRi. Consequently, by Lemma 3.5), for any e' 0, there exists 
sequence of mapping pairs 

(/i : Xr' ^ Wi, ^1 : Wi X Z^' ^ Zf) 

(for some n' —>■ 00) such that (3.21) and (3.22) hold. In other words, we respectively have 
((3.21) is divided throughout by n) 

^log|Wi| < Ri + e'l/n (5.4) 
nn 

PT{{xr\z;',z;')^Ty\xiz2,z,)} < e'l (5.5) 

where Z^ = g(i(/i(X™'), Z2 ) and e'( — > as e' — > 0. Further, denote 

X^ij) = V'(Zi(j),i2(j)), i<j<n' 

xf = i^'iz';',zt)^g{Mxr'),f2{xr')). 

Clearly, g :Ui XU2 ^ A"""'. Due to (5.5), we have (adjusting n' — > 00, if necessary) 

Pr{^i} < 6^' (5.6) 

where 

= {(xr',V''(^f ,^2') ^ Ty\x^,^{Z,,Z2))} 
and 62 — > as e' ^ 0. Since 

nn' 

for any (x^',^?"') e ^''"'^(^i", "0(^1, ^2)) due to (2.18), we have 

< (l-Pr{^i})(D + e'rfniax) 

< D + {e' + e'^d^^ (5.7) 



-^Edr,n'{Xr',rr') < (l-Pr{^i})(D + e'rfniax) + Pr{^i}4 
nn' 



due to (5.6). 

Now, for a particular e > 0, choose e' such that max{e", e'l, (e' + e2)c?max} < e- Hence con- 
ditions (5.4), (5.1) and (5.7) give rise to conditions (2.4), (2.5) and (2.14), where (^1,^2, nn') 
now take the place of {Zi,Z2,n). Hence {Ri,R2,D) e Aop- In other words, 2 «4.*p. 
Since is closed, we have 2 «4*p (noting is the smallest closed set with as 
a subset). This completes the proof. □ 
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5.2 Outer Bound A^p C A*^ 

Recall that any triplet {Ri, R2,D) e if, for any e > 0, there exists partial side informa- 
tion code 

of some length n such that (2.4), (2.5) and (2.14) hold. The above conditions are reproduced 
below for easy reference: 

-log|Zi| < i?i + e (5.8) 
n 

-log 1 < i?2 + e (5.9) 
n 

-Ed„(Xr,^(/i(Xn,/2(^2"))) < D + e. (5.10) 

We can further encode Zi = fi{X^) with complete side information Z2 = 72(^^2 ) using 
Slepian-Wolf code 

Given {Ri,R2,D) e ^d, e and (/i,/2,5'), rate R[ is said to be achieved using interposed 
Slepian-Wolf codes if for any e' > there exists (/{, g[) (of length n') such that 

^log|Wi| < R[ + e' (5.11) 
n' 

Pr{Z^' ^g[{f[{Z^'),Z^')} < e'. (5.12) 

In view of (5.8), setting /{ to identity mapping (clearly, n' = 1, Ui = Z\) and choosing 
^i(/{(Zf),Z2"') = /U^r') = ^r', of course, (5.11) and (5.12) trivially hold for i?; = n(i?i + e) 
irrespective of e'. Therefore, we have 

^i/(/i(Xr)|/2(X2"))<i?i + e (5.13) 

by Slepian-Wolf theorem [10]. In the same manner, we can also encode Z^ — j^i^^) using 
an interposed Shannon code 

{r^:Zl-^U2.g'^:U2^Zl-). 

Again given (i?i, R2,D) e Ao, e and (/i, f2, g), rate R2 is said to be achieved using interposed 
Shannon codes if for any e' > there exists (/2, g'2) (of length n') such that 

^log|W2| < R2 + e' (5.14) 
n 

Pt{Z^' ^ g'MZ^'))} < e'. (5.15) 
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In view of (5.9), setting each of and g2 to identity mapping (clearly, n' — 1, U2 — Z2), of 
course, (5.14) and (5.15) trivially hold for R2 — n(i?2 + e) irrespective of e'. Therefore, we 
have 

-H{f2iX^))<R2 + e (5.16) 

Th 

by Shannon's lossless coding theorem [10]. 

At this point define for any e > and any integral n > 1 the set ^*/dp of rate-distortion 
triplets {Ri, R2, D) such that there exist alphabets Zi and Z2, conditional distributions 
qi{zi\xi) and 52 (^2! 2:2), and mapping ip : Zi x Z2 ^ x A"^, satisfying 

-/(Xf;Zi|Z2) < Ri + e (5.17) 

Th 

-I{X^-Z2) < R2 + e (5.18) 

n 

-dn{X^,ilj{Zi,Z2)) < D + e (5.19) 
n 

where {X^, X^, Z,, Z2) ~ Pn{x'l,x^)qi{zi\x'^)q2{z2\x^) (i.e., Zi ^ X^ ^ Z2 is a 

Markov chain) and Pn{x1,X2) = nfc=iP(^i(^)> ^2(^))- Comparing with (2.16)-(2.18), note 
that A*}^1 = A*^^^. In addition, ^1%^ C A*}^^ for ei < e2_^rther, let = U"=i ^^nol 
Of course, ^dp^^ Q «4dp^^ for ei < 62. Hence, noting Pj^^g^n^p^ is closed, we obtain 

n^ = ^ = :4*;. (5.20) 

e>0 

The second equality in (5.20) holds because ^dp^ = U^i-^nop = U^^i-^nop = -^dp- 

Finally, consider any {Ri, R2, D) G Aup- Recall that for any e > there exists partial 
side information code (/i, f2,g) such that (5.13), (5.16) and (5.10) hold. Choosing {Zi, Z2) = 
(/i(X"), /2(X2 )) (of course, Zi X" Z2 is then a Markov chain) and ip = g, note 

that the above three conditions coincide with (5.17)-(5.19), respectively. Hence {Ri, R2, D) G 
J^^p. Consequently, we have {Ri, R2, D) G ^dp"* C Aop for each e > 0. Hence, by (5.20), 
we have {Ri, R2, D) G n€>0"^Dp'' = ^dp- This completes the proof. □ 



6 Known Coding Theorems 

So far we have solved (in Theorems 2.1 and 2.2, respectively) the joint distortion and partial 
side information problems which hitherto remained open. More precisely, we have outlined a 
new methodology that expands older techniques and solves the above open problems. We now 
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illustrate that our technique applies to the known problems equally well. In particular, we 
now outline based on our method proofs of four widely celebrated results, namely. Shannon's 
lossy coding theorem [2], Wyner-Ziv theorem [6], side information theorem [4, 5] and Berger- 
Yeung theorem [9], in Sees. 6.2-6.5, respectively. First we compare our technique with its 
classical counterpart in Sec. 6.1. 

6.1 Comparison with Classical Proofs 

Corresponding to each known problem under consideration, we shall define the achievable 
rate (rate-distortion) region A. As seen in Sec. 2, we shall also define rate (rate-distortion) 
regions ^* that have suitable n-th order information-theoretic descriptions and let A* — 
\Jn=i^n- We subscript these sets by 'S', 'WZ', 'SF and 'BY', respectively, to indicate the 
correspondence with the known theorems listed above. We know that each theorem describes 
the corresponding achievable region by a first order description of the form A = Al [10, 9]. 

The classical proof of each of the above theorems consists of two steps: Proof of the direct 
theorem A ^ A^ and proof of the converse A A^. Clubbing n source symbols together 
to start with (where n is arbitrarily chosen), one can of course show A 2 A^ and A C ^* 
using the same argument, thus in effect showing (since Al is closed) 

A — A* — A* — A* 

✓T. ✓T. ^rX ' 

In fact, we have discovered (further details can be found in [12]) that the coding theorems 
corresponding to a broad class of problems (including, of course, the above known results as 
well as Theorems 2.1 and 2.2) take the form A = A*. Accordingly, we prove such theorems 
by showing the inner bound A^ A* and the outer bound A Q A*. Subsequently, we carry 
out the simplification A* = Al wherever possible. Clearly, our strategy yields the classical 
solution for the known problems where we indeed have A* — Al- However, when such 
simplification does not arise (as seen, for example, in Theorems 2.1 and 2.2), the classical 
technique fails. 

Proceeding in the same manner as in case of Theorems 2.1 and 2.2, we can show the inner 
bound AQ A* for the known cases using the fundamental principle of source coding (given 
in Theorem 3.1 in the general setting and in Lemmas 3.2 and 3.5 in special cases). Now, from 
Sees. 2.2 and 2.3, recall that Theorems 2.1 and 2.2 deal with encoding under a distortion 
criterion and do not involve lossless coding. Also, from Sees. 4.2 and 5.2, recall that, to 
show the outer bound ^ D ^* in the above cases, we required Shannon's lossless coding 
theorem and Slepian-Wolf theorem. Subsequently, we shall see that similar techniques will 
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apply in case of Shannon's rate-distortion and Wyner-Ziv theorems as well, neither of which 
involves lossless coding. However, side information and Berger-Yeung theorems deal with 
problems where one source is losslessly decoded. In such cases, we shall additionally need 
Fano's inequality [10]. 

Finally, consider the proof of A* — Al in the four known cases at hand. Since 

oo 

a^da* = \Ja*^^ai, 

n=l 

it is enough to show ^* C Al for arbitrary n. In fact, this has been shown in Appendix A in 
case of Wyner-Ziv theorem (read subscript 'DC as 'WZ'). The proof makes use of convexity 
of Al and reproduces crucial steps from the classical proof of Wyner-Ziv's converse theorem. 
Presenting this as an illustration, we omit the respective proofs of A^ Q Al corresponding 
to the other known problems. The reader with an eye for details is encouraged to construct 
such proofs by referring to classical proofs of converse statements of corresponding known 
theorems and identifying crucial steps. 



6.2 Shannon's Lossy Coding Theorem 

Formalism: Consider encoding of X ~ p{x) using encoder mapping / : X"- — > Z and 
decoding using decoder mapping g : Z ^ under distortion criterion d : X"^ ^ [0, cimax]- 
A rate-distortion pair (i?, D) is said to be achievable if for any e > there exists (for n 
sufficiently large) mapping pair (/, g) such that 

-loglZl < R + e (6.1) 
n 

-EdJX'',X'') < D + e (6.2) 
n 

where — g{f{X'^)). Denote the set of achievable pairs {R, D) by ^g. 

Solution: Let be the set of (i?, D) pairs such that there exist alphabet Z C A"" and 
conditional distribution q{z\x^) satisfying 

-/(X";Z) < R (6.3) 
n 

-dn{X^,Z) < D. (6.4) 
n 

Moreover, denote Al — U^i-^ns- Then Shannon's lossy coding theorem states As — Al^ 
[2], whose proof according to our methodology consists of three steps showing D Al, 
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As Q Al and Al — Ai^, respectively. In the following, we sketch the first two steps of the 
proof and skip the third as indicated before. 

Proof of Inner Bound: To show the inner bound ^ ^® generalize the argument 
(given immediately following the statement of Lemma 3.2) used in sketching the proof of 
As 2 Als- For any {R,D) e At = U^=i'4ns> by definition, (6.3) and (6.4) hold for some 
n. Referring to Lemma 3.2 and identifying (F, Z) with (X", Z) and R' with nR, note that 
(6.3) is same as (3.7). Consequently, by Lemma 3.2, for any e' — > 0, there exists sequence of 
mapping pairs (/ : A"""' — > ZY, : W — > Z"') (for some n' oo) such that (3.8) and (3.9) hold. 
Due to (3.8), ^ log \U\ is close to R, which is same as the condition (6.1), with (W, nn') now 
playing the role of {Z,n). Further, by (3.9), the reconstruction = g{f{X"-'^')) is jointly 
typical with X""' (the joint typicality being with respect to {Z,X'^)) with high probability. 
Hence, now dubbing the above reconstruction as X , distortion ^E(i„„'(X"" , X ) is 
close to D due to (6.4), which is same as condition (6.2) with nn' in place of n. Therefore, 
we have (i?, D) e As-, implying As ^ Al,. Further, since .4s is closed, we obtain .As 5 v4.|. 

Proof of Outer Bound: To show the outer bound As ^ A^-, consider any (i?, D) G As- 
By definition, for any e > there exists mapping pair (/ : A*" Z^g : Z ^ X"-) of 
sufficiently large length n such that (6.1) and (6.2) hold. Now, in view of (6.1), using 
interposed Shannon coding of /(X"), one can show (in the spirit of Sees. 4.2 and 5.2) 

-H(g(f{X^)))<-H(f(X")) < R + e. (6.5) 
n n 

Finally, for the choice Z = g{f{X^)), (6.5) and (6.2) become (6.3) and (6.4), respectively, 
except for e added to each right hand side. Using an argument similar to that given in 
the last two paragraphs of either of Sees. 4.2 and 5.2, one can show {R,D) e A^- Hence 
As QA*. 



6.3 Wyner-Ziv Theorem 

Formalism: Consider random variables {Xi,X2) ~ p{xi,X2)- Encode Xi using encoder 
mapping fi : Zi and decode using complete side information X2 and decoder mapping 

g : Zi X X2 X]^ under distortion criterion d : X^ ^ [0,rfmax]- A rate-distortion pair 
{Ri,D) is said to be achievable if for any e > there exists (for n sufficiently large) a 
mapping pair {fi,g) such that 

-log 1^1 1 < Ri + e (6.6) 
n 

-Ed„(X^Xi) < D + € (6.7) 
n 
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where — g{fi{X'^), X2). Denote the set of achievable pairs {Ri, D) by ^wz- 

Solution: Let A^^^ be the set of (i?, D) pairs such that there exist alphabet Zi, condi- 
tional distribution qi^Zilx"^) and mapping ip : Zi x Xl\ satisfying 

-I{X^;Zi\X^) < Ri (6.8) 

-dn{Xl'i/j{Zi,X^)) < D (6.9) 
n 

where {X^,X^,Zi) ~ pn{x'l,x^)qi{zi\x'l) (i.e., Zi X^ ^ X^ is a Markov chain) and 
Pn{x'l,x'^) = Ylk=iPi^dk),X2{k)). Further, denote Al,^ = [Jn=i^nwz- Then Wyner-Ziv 
theorem states A^f,z — ^iwz [6] ; whose proof according to our methodology consists of three 
steps showing A^z 2 A^^^ ^^wz Q ^wz Al,^ = >^iwz' respectively. In the following, we 
sketch the first two steps of the proof and skip the third as indicated before. 

Proof of Inner Bound: To show the inner bound Awz 5 >^wz' generalize the 
argument (given immediately following the statement of Lemma 3.5) used in sketching the 
proof of A^z ^ ^iwz- For any {R, D) G ^^z = Ur=i'^nwz) by definition, (6.8) and (6.9) 
hold for some n. Referring to Lemma 3.5 and identifying {Yi, Y2, Zi) with (X", X^, Zi) and 
R'^ with nRi, note that (6.8) is same as (3.20). Consequently, by Lemma 3.5, for any e' — > 0, 
there exists sequence of mapping pairs (/i : X'-'"''' l^i, g '■ Ui x ^2^' Z^') (for some 
n' ^ 00) such that (3.21) and (3.22) hold. Due to (3.21), ^ log \Ui\ is close to Ri, which is 
same as the condition (6.6) with (Wi, nn') now playing the role of {Zi, n). Further, by (3.22), 
if = ^(/i(X{'"'),X^"') is jointly typical with (X{^"', X^'^') with high probability. Hence, 
X^"^' coupled with the reconstruction 

xT' = (V'(^i(l),X2"(l)),z^(Zi(2),X,"(2)),...,V(^i(n'),X2"(n'))) 

is a jointly typical sequence of (X", ^(Zi, X2 )) with high probability, implying, in view of 
(6.9), that the distortion ^E(i„/(X""', ) is close to which in turn is same as condition 
(6.7) with nn' in place n. Therefore, we have (i?, D) e ^wz, implying A^^fz ^ ^wz- Further, 
since A^^z is closed, we obtain A^^z 2 A%j^. 

Proof of Outer Bound: To show the outer bound A^^z Q A%j^, consider any (i?i, D) e 
A^Nz- By definition, for any e > there exists mapping pair (/i : Zi, g : Zi x X^ — > 

X^) of sufficiently large length n such that (6.6) and (6.7) hold. Now, in view of (6.6), using 
interposed Slepian-wolf coding of /i(X") with X^ as the complete side information, one can 
show (in the spirit of Sees. 4.2 and 5.2) 

-E{j^{Xl)\Xl) < R, + e. (6.10) 
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Now, for the choice Z]_ — j\{X'^^ (of course, — > X"^ — > is a Markov chain) and ip — g, 
(6.10) and (6.7) become (6.8) and (6.9), respectively, except for e added to each right hand 
side. Using an argument similar to that given in the last two paragraphs of either of Sees. 4.2 
and 5.2, one can show {R, D) e ^^z- Hence Q ■^%,z- 



6.4 Side Information Theorem 

Formalism: Consider lossless encoding using partial side information. Specifically, encode 
(Xi,X2) ~ p{xi^X2) using encoder mapping pair (/i : — ^ Zi, /2 : X2 Z2) and decode 
Xx losslessly (in the sense of Shannon) using decoder mapping g : Zx ^ ^ X^. A rate 
pair (i?i, R2) is said to be achievable if for any e > 0, there exists (for n sufficiently large) a 
mapping triplet {fi,f2,g) such that 

-loglZil < Ri + e (6.11) 

n 

-log|Z2| < R2 + e (6.12) 
n 

Pr{Xi"^x"} < e (6.13) 

where Xi — g{fi{Xx), /2(X2 )). Denote the set of achievable pairs (i?i, R2) by Asi- 

Solution: Let ^*gj be the set of (i?i,i?2) pairs such that there exist alphabet Z2 and 
conditional distribution q2{z2\x2), satisfying 

-H{X^\Z2) < Ri (6.14) 
n 

-I{X^-Z2) < R2 (6.15) 
n 

where {X1,XI^,Z2) ~ Pn{x1,xf)q2{z2\xf) (i.e., X'l ^ Xl^ ^ Z2 is a Markov chain) and 
Pn{x1,X2) — 11^=1 ^2(^))- Further, denote = U^i'^nsi- Then side information 

theorem states = ^^gj [4, 5], whose proof according to our methodology consists of three 
steps showing D C Al^ and Al^ — A\^^, respectively. In the following, we sketch 

the first two steps of the proof and skip the third as indicated before. 

Proof of Inner Bound: For any (i?i, R2) G ^*si, by definition, (6.14) and (6.15) hold 
for a Markov chain X^ — > X2 — > Z2. Now, consider Theorem 3.1 and recall from Sec. 3.4 
that any rate pair {R[, R2), such that 

R[ > I{Yx,Zx\Z2) (6.16) 
R!2 > I{Y2;Z2), (6.17) 
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also satisfies (3.1)-(3.3). Further, identifying (Fi, Fa, ^i, ^2) with {X'l , , , Z2) (note 
/(Xf ; Z1IZ2) = IZ2) because = ) and (i?;, i?Q with ni^a), note that (6.14) 

and (6.15) are same as (6.16) and (6.17), respectively. Hence, (3.1)-(3.3) too hold. Conse- 
quently, by Theorem 3.1, for any e' — > there exists a sequence of mapping triplets (/i, /2, g) 
such that (3.4)-(3.6) hold. Further, due to (3.4) and (3.5), ^log|Wi| and ^log|W2| are 
close to Ri and it!2, respectively, which in turn are same as the respective conditions (6.11) 
and (6.12) with {Ui,U2,nn') now playing the role of (^1,^2,71). Recalling Zi — X" and 
denoting X^ — ^\ (3.6) imphes that X"" differs from X^ with low probability, which is 
condition (6.13) with rml in place of n. Hence (i?i, R^) G ^si, implying 2 Further, 
since is closed, we obtain 2 

Proof of Outer Bound: The proof requires Fano's inequality (as does the classical 
proof of the direct statement of side information theorem), a weakened version of which 
states the following [10]: Given random variables U and V, 

H{U\V) <l + \og\U\Vi{U ^ giV)} (6.18) 

for any g : V ^ U. Now consider any R2) G ^si- By definition, for any e > there exists 
(for n sufficiently large) mapping triplet (/i : ^ Zi, f2 : X2 ~^ ^2,g : Zi x Z2 ^ X^) 
such that (6.11)-(6.13) hold. Now, in view of (6.11), using interposed Slepian-wolf coding of 
/i(X") with /2(X2 ) as the complete side information, one can show 

R, + e > -H{f,{X^)\f2{X^)) (6.19) 
n 

= -H{X^J,{X^)\f2{Xl^)) - -H{X^\f,{X^)j2{Xl^)) 
n n 

= -H{X^\f2{Xli))--H{X^\h{X^)j2{Xli)). (6.20) 
n n 

Since Vi{X'^ 7^ g{fi{X1)j2{X'2 ))} < e by (6.13), we have, by Fano's inequality (6.18), 

H{X^\f^{X^),f2{X^)) <l + nelog\X,\<ne{l + log\X,\) (6.21) 

where the second inequality can be ensured by choosing n > 1/e. Using (6.21) in (6.20), we 
obtain 

-H{X]^\f2{X^))<Ri + ei2 + \og\X^\). (6.22) 

Th 

Further, in view of (6.12), one can show, using interposed Shannon coding of /2(X^), that 

-H{f2{X^))<R2 + e. (6.23) 
n 

Now, for the choice Z2 = 72(^2) (clearly, X]" X^ Z2 forms a Markov chain), (6.22) 
and (6.23) become (6.14) and (6.15), respectively, except for a multiple of e added to each 
right hand side. Using an argument similar to that given in the last two paragraphs of either 
of Sees. 4.2 and 5.2, one can show (i?i, R2) G Hence Q Al^. 
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6.5 Berger-Yeung Theorem 

Formalism: Consider encoding {Xi,X2) ~ p{xi,X2) using encoder mapping pair (/i : 
A"" Zi, f2 : ^2 Z2) and decoding using decoder mapping g : Zi x Z2 ^ x X^- 
For the sake of convenience, we write g = {gi,g2), where gi and g2 has ranges that are 
subsets of A"" and X2, respectively. In particular, gi decodes Xi losslessly (in the sense of 
Shannon) whereas g2 decodes X2 under a bounded distortion criterion d : X2 [0,(imax]- 
A rate-distortion triplet i?2, D) is said to be achievable if for any e > 0, there exists (for 
n sufficiently large) a mapping triplet {fi,f2,g) such that 

-loglZil < Ri + e (6.24) 
n 

-log 1 < R2 + e (6.25) 

n 

Pr{Xi"^x"} < e (6.26) 

-Edn{X^,X2) < D + e (6.27) 

where (X^^X^) = 5f(/i(X"), /2(X2 )). Denote the set of achievable triplets {Ri, R2, D) by 

Solution: Let v4*j3y be the set of (i?i, R2, D) triplets such that there exist alphabet Z2, 
conditional distribution 52 (2^2 12^2) mapping -0 : x Z2 ^ X2, satisfying 



-H(X'^\Z2) < Ri (6.28) 
n 

-IiX^;Z2\X^) < R2 (6.29) 
n 

H{Xi) + -I{X^;Z2\X'^) < R1 + R2 (6.30) 

-dniX^,ip{XlZ2)) < D (6.31) 



where (X]^,X^, Z2) ~ Pn{xi,X2)q2{z2\x2) (i.e., X^ ^ X^ ^ Z2 is a Markov chain) and 
p„(x^,x5) = nfe=i^'(^i(^)'^2(^))- Further, denote Al^ = U^i^neY- Then Berger-Yeung 
theorem states ^by = «4iby [9] > whose proof according to our methodology consists of three 
steps showing ^by ^ A^^., Aby Q A%y A%y — AIj^yj respectively. In the following, we 
sketch the first two steps of the proof and skip the third as indicated before. 

Proof of Inner Bound: For any (i?i, R2, D) e .4*by, by definition, (6.28)-(6.31) hold 
for some and some Z2 such that X" — > X2 — > Z2 is a Markov chain. Now, referring to The- 
orem 3.1, let us identify (Fi, Y2, Z^, Z2) with (X^", X^, , Z2) and {R[, R'2) with (ni?i, ni?2)- 
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Noting I{X^;Zi\Z2) = H{X^\Z2) (because = X^), (6.28) is same as (3.1). Also, (6.29) 
coincides with (3.2) in a straightforward manner. Next, in view of equahty (3.47), we have 

/(Xr, Xl\ Z2) = /(Xf; + /(X^^ ZalXf) = nH{X,) + I{X^; Z2\Xl). 

Hence (6.30) coincides with (3.3). Consequently, by Theorem 3.1, for any e' — > there exists 
a sequence of mapping triplets (/i,/2,g') such that (3.4)-(3.6) hold. Further, due to (3.4) 
and (3.5), ;^log \Ui\ and ^log|W2| are close to Ri and ii!2, respectively, which in turn are 
same as the respective conditions (6.24) and (6.25) with (^1,^2, nn') now playing the role of 
(2^1, ^2,^). Write g — {91,92} for convenience, where ranges of gi and g2 are subsets of 
and 2^2 ', respectively. RecaUing Zi — X" and denoting 

xr' = ^r' = ^?i(/i(xr'),/2(xr')), 

(3.6) implies that Xj^*^' differs from X^^ with low probability, which is condition (6.26) 
with nn' in place of n. Again, by (3.6), Z^ = g2{fi{X^"' ), /2(X^'* )) is jointly typical with 
(X^ , X2 "') with high probability. Hence, X^"' coupled with the reconstruction 

xf = V''(^rU;') = (V'(^r(i),^2(i)),V'(^r(2),i2(2)),...,v(^r(n'),^2(n'))) 

is a jointly typical sequence of {X2,ip{X^, Z2)) with high probability. Hence, the distortion 

^E4'(^r',^r'), where 

xf = i^'{xT\ zi) = i,\g{h{Xf\ h{XT'))) = 9\h{XT\h{XT')\ 

is close to D due to (6.31), which in turn is same as condition (6.27) with {nn',g') in place 
of {n,g). Hence (i?i,i?2,-D) G Aby, implying ^ ^by- Further, since ^by is closed, we 
obtain ^by ^ -Aby- 

Proof of Outer Bound: Consider any (i?i,i?2) G -4by- By definition, for any e > 
there exists mapping triplet (/i : A"" — > Zi, f2 : X2 ^ Z2, g : Zi x Z2 ^ X'l x X2 ) of 
sufficiently large length n such that (6.24)-(6.27) hold. Now, in view of (6.24) and (6.25), 
using interposed Slepian-wolf coding of (/i(X{'), /2(X2 )) (as seen in Sec. 4.2), one can show 

-H{f,{X^)\f2{X^)) < Ri + e (6.32) 

n 

-H{f2{X^)\fi{X^)) < R2 + e (6.33) 
n 

-//(/i(Xi"),/2(X-)) < i?i + i?2 + 2e. (6.34) 

n 

Further, since Pr{Xf ^ g,{fi{X^), f2{X^))} < e (recall g = (^1,^2)) by (6.26), we have, by 
Fano's inequality (6.18), 

H(X^\f,(X^),f2(X^)) < l + neloglA-il <ne(l + log|A'i|) (6.35) 
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where the second inequahty can be ensured by choosing n > 1/e. At this point, noting 
conditions (6.32) and (6.35) to be same as conditions (6.19) and (6.21), respectively, we 
reproduce (6.22) below: 

^H{X^\h{Xl^)) <R, + e(2 + log \X,\). (6.36) 

Also, noting /i(Xf) ^ f2{X^) is a Markov chain, we have H{f2{X^)\X^) < 

H{f2{X^)\fi{Xl^)). Therefore, from (6.33), we have 

-H{f2{X^)\X^)<R2 + e. (6.37) 
n 

Moreover, write (6.34) as 

R^ + R2 + 2e > ^H{f,{X^), MX^)) 

= -H{X^, MX^), MX^)) - -H{X^\MX^), h{X^)) 
n n 

= -H{x^, Mx^)) - -H{x^\Mx^), Mx^)) 

n n 
= H{X,) + -H{f2{X^)\X^)-e{l + \og\X,\) (6.38) 

Th 

where (6.38) follows using (6.35). Rearranging (6.38), we have 

H{Xi) + -H{f2{X^)\X^) <R, + R2 + e(3 + log (6.39) 
n 

Further, from (6.27), write 

D + e > -Edn{Xlg2{MX[^)j2{X;^))) 

Th 

1 -^n 

= -Edn{Xlg'2{X„MX^)j2{X^))) (6.40) 

Th 

> (1 - Pr{X- + X\^)^d^{Xl,g'2{Xl,h{Xl),f2{Xl))) 

> -EdniX^ g',{X^, MX^), f2{X^))) - edmax (6.41) 

n 

= -E4(X», iPiX^, f2{X^))) - 6(imax (6.42) 
n 

where (6.40) holds for suitable g'2, (6.41) follows because Pr{Xf ^ x"} < e (by (6.26)) and 
(6.42) holds for suitable V'- Rearranging (6.42), we have 

-Edr^iXl iPiX^, f2{X^))) < e(l + dmax). (6.43) 

Th 

Now, for the choice Z2 — /2(-'^2) (o^ course, — > X2 — > is a Markov chain), (6.36), 
(6.37), (6.39) and (6.43) become (6.28)-(6.31), respectively, except for a multiple of e added 
to each right hand side. Using an argument similar to that given in the last two paragraphs 



of either of Sees. 4.2 and 5.2, one can show (i?i, i?2, D) e A%-^. Hence Q Al 
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7 Conclusion 



In this paper, we presented a unified solution methodology that solves any two terminal 
source coding problem where individual encoders do not cooperate. In particular, we solved 
using our method the joint distortion and the partial side information problems which re- 
mained hitherto open. More generally, we have shown in our analysis that the achievable 
rate-distortion region in any two terminal problem admits an infinite order information- 
theoretic description. We also note that simplifications arise in certain special cases which 
have been known to have first order solutions. Summarizing, the principal contribution of 
our paper lies not in providing solutions to individual source coding problems but in iden- 
tifying a fundamental principle (Theorem 3.1) of source coding arising out of the notion of 
typicality. In fact, this central principle captures and extends the basic typicality arguments 
of Shannon [2] and Wyner-Ziv [6]. In our proofs, we also made extensive use of interposed 
lossless coding, which again was conceived by Wyner-Ziv. In this connection, we in turn 
needed Shannon's [1] and Slepian- Wolf's [3] lossless coding theorems. In other words, we 
picked the available building blocks from existing techniques, organized them appropriately 
and built on them further to develop a unified framework for two terminal source coding. In 
the second paper of this two part communication [12], we shall see that our unified frame- 
work extends to problems with any number of sources. In particular, our methodology will 
solve all multiterminal source coding problems where individual encoders do not cooperate. 
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A Wyner-Ziv Theorem as Corollary 



Recall that, in order to complete the derivation of Wyner-Ziv theorem as a corollary of 
Theorem 2.2, we are left to show two facts: 

1. For a given n, A^^^ = {{Ri, D) : (i?i,7?2,^) e A*dp>^2 = ^^(^2)} is the set of 
{Ri,D) pairs such that (2.21) and (2.22) hold for some q2 and ip. 

2. For any n, ^ -^^dc- 

We show the above in Appendices A.l and A. 2, respectively. 

A.l Description of ^'^^^ 

In Sec. 2.3, we have seen that A^oc is given by the set of (i?i, D) satisfying (2.16) and (2.18) 
for some Markov chain Zi — > X" — > X2 Z2. However, noting 

{(i?i,D) : iRi,R2,D) e Al^,} D {iRi,D) : iR^,R'2,D) e Al,,} 

for R2 > R2, we can and shall include, without loss of generality, the additional condition 

-I{X^;Z2)^H{X2) (A.l) 
n 

in the description of ^dc- Note that (A.l) imphes X2 = ^(-^2) with probability one for 
some function 0, i.e., Xi Z2 ^ X2 is a Markov chain. At the same time, recall from 
Sec. 2.3 that we require X" — > X^ Z2 to form a Markov chain as well in the description 
of AnY)c- However, the above two Markov chains can simultaneously occur in only two 
circumstances: 1) X'^ is independent of {X^, Z2), and/or 2) Z2 = X^ with probabihty one. 
The first possibility is ruled out assuming Xi and X2 are statistically dependent. Hence, 
accepting the second possibility and referring to (2.16) and (2.18), w4*n(. has the desired 
description. 

A.2 Proof of A:,^ C A^,^ 

The proof in essence reproduces part of the classical proof of Wyner-Ziv's direct theorem 
and assumes convexity of ^iqc- Verify that such convexity is equivalent to convexity of the 
function 

min i?i 

D:{Ri,D)eAl^C 
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whose proof appears in [6] and is self-contained (thus avoiding circular argument). 
For any pair {Ri,D) G A*^oc, (2.21) and (2.22) hold. Next from (2.21), write: 

Ri > -i{x^]Zr\xi^) 

Th 



1 

= -E^(^i(^);^ii^2,^i-') (A.2) 

fe=i 

= -J2H{X^(k)\X^,Xt') --j2H(X^(k)\X^,Xt\Z,) 

k=\ k=l 

- ^j2HiMk)\X2{k)) 

k=l 

1 " 

— y2H{X,{k)\Xt\X2{k),X2{k + l;n),Xt\Z,) (A.3) 

k=\ 

> -y2H{Xr{k)\X2{k)) - -y2H{Xr{k)\Xt\X^{k),X2{k + l;n),Zr) (A.4) 

k=l k=l 

1 

= -Y,I{Mk);Z[{k)\X,{k)l (A.5) 



n 

k=l 



where (A.2) follows by chain rule, (A.3) follows because Xi(k) is independent of ^ and by 
writing X2" = {Xt\X2{k),X2{k+l;n)), (A.4) follows because conditioning reduces entropy, 
and (A.5) follows by denoting Z[{k) = {X2~^,X2{k + l;n),Zi), 1 < k < n. Further, from 
(2.22), we can write 

D >-J2MMk),i^kiZi,x^)) ^ -J2MMk),i^kizm,X2{k))), (a.6) 

Th Th 

k=l k=l 

where 'ip'^: ZiX X2 Xi denotes the mapping that produces the /c-th symbol produced by 
-0', and the mapping ■0|^'((X2~\ X2(A; + 1; n), Zi),X2{k)) = ■?/'^(Zi, ) simply rearranges the 
arguments of ip'j^. Comparing (A.5) and (A.6) with (2.21) and (2.22), respectively, and noting 
that Z[{k) Xi{k) — > X2{k) is a Markov chain for each 1 < /c < n, we have (i?i, D) e ^1^0 
due to convexity of A\^q. Hence the result. □ 
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